{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 20 - Hierarchical Clustering\n",
    "\n",
    "We will look at two datasets today as we study hierarchical clustering.  \n",
    "\n",
    "## Clustering the Iris data\n",
    "\n",
    "The first is the [iris dataset](https://en.wikipedia.org/wiki/Iris_flower_data_set), which contains 50 samples each of 3 types of irises (Iris setosa, Iris virginica and Iris versicolor).  The 4 measurements for each iris are the length and width (in cm) of the [sepals](https://en.wikipedia.org/wiki/Sepal) and petals.\n",
    "\n",
    "The iris dataset is included in the sci-kit learn package, so we can load it from there."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "import scipy.cluster.hierarchy as shc\n",
    "\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "\n",
    "from sklearn.cluster import AgglomerativeClustering\n",
    "\n",
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "from sklearn import datasets\n",
    "\n",
    "%matplotlib inline\n",
    "pd.set_option(\"display.max_columns\", None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As with the Boston housing dataset, we can load the iris dataset from sci-kit learn.  The iris dataset is also in dictionary format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris_dict = datasets.load_iris()\n",
    "iris_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Type `iris_dict.keys()` to see what is included in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a dataframe from the dictionary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris = pd.DataFrame(iris_dict.data, columns = iris_dict.feature_names)\n",
    "iris.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display your new dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we will compare the rows using the Euclidean distance, we should scale all columns to be between 0 and 1.  Do this below, storing the scaled data in the variable `iris_scaled` (see Lab 19)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we will plot a dendrogram (tree) of the distances between the irises."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10, 7))  \n",
    "plt.title(\"Dendrograms\")  \n",
    "dend = shc.dendrogram(shc.linkage(iris_scaled, method='average'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can only plot the tree with the scipy package, so we need to use its linkage (clustering) function to produce the tree in a compatible format.\n",
    "\n",
    "However, it is easier to use the sklearn package implementation for all other analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster = AgglomerativeClustering(n_clusters=3, affinity='euclidean', linkage='average')  \n",
    "clusters = cluster.fit_predict(iris_scaled)\n",
    "clusters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add the clusters to the `iris` dataframe as a new column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To see how our clustering did, let's create a scatter plot of two of the columns in iris (your choice of columns), colored by the cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How well do you think the clustering algorithm worked from this graph?\n",
    "\n",
    "Let's compare it with the same plot, colored by the true type of iris.\n",
    "\n",
    "First, add a new column to the `iris` dataframe with the `target` data in the dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, plot the same two variables as above, colored by the true type of iris."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How do your two plots compare?  What happens if you try a different pair of variables?\n",
    "\n",
    "We can also use a confusion matrix to compare the predicted clusters with the real ones.  Try it below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is the accuracy of this clustering method?\n",
    "\n",
    "What happens if you try Ward linkage instead?\n",
    "\n",
    "### Clustering labor market data\n",
    "\n",
    "The Federal Reserve Bank of New York has information about the labor market for recent college graduates [here](https://www.newyorkfed.org/research/college-labor-market/college-labor-market_compare-majors.html).\n",
    "\n",
    "The data in this table can be downloaded as an Excel file at the bottom of the page. If you open this file in Excel, you can save the last table as a CSV file. Alternatively, download the data as a CSV file from the course website.\n",
    "\n",
    "Open the CSV file in Jupyter or another text editor to see if there are extra lines that need to be accounted for when reading it in.  Recall that `read_csv()` has the optional parameters `skiprows` and `skipfooter` (ignore the warnings this parameter generates).\n",
    "\n",
    "Additionally, set the index to be the `Major` column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall we can look at the types of the columns using the pattern `df.dtypes`.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "labor.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Which two columns are not numerical types (integers or floats)?  Can you guess why?\n",
    "\n",
    "The following code removes the commas and converts the type to float."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "labor[\"Median Wage Early Career\"] = labor[\"Median Wage Early Career\"].str.replace(\",\",\"\").astype(float)\n",
    "labor[\"Median Wage Mid-Career\"] = labor[\"Median Wage Mid-Career\"].str.replace(\",\",\"\").astype(float)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check that the columns all have a numerical type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scale the data in each column to be between 0 and 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can put the scaled data back into a dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "labor_scaled = pd.DataFrame(labor_scaled, columns = labor.columns, index = labor.index)\n",
    "labor_scaled"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the dendrogram using Ward linkage.  To label the leaves as the majors, add the parameter `labels = labor_scaled.index` to the `dendrogram()` function.  \n",
    "\n",
    "You might also want to increase the leaf font size using the optional parameter `leaf_font_size`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do you notice about the tree?  Do the relationships of the leaves make sense?\n",
    "\n",
    "Let's compute the cluster using sklearn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add the predicted clusters as a column to the `labor` dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the clusters by using a filter to display only the rows in that cluster.  Do the clusters make sense?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}